Method and system for snapshot multi-spectral light field imaging

ABSTRACT

A method for generating high resolution multi-spectral light fields is disclosed. The method may include capturing a multi-perspective spectral image which includes a plurality of sub-view images; aligning and warping the sub-view images to obtain low resolution multi-spectral light fields; obtaining a high resolution dictionary and a low resolution dictionary; obtaining a sparse representation based on the low resolution multi-spectral light fields and the low resolution dictionary; and generating high resolution multi-spectral light fields with the sparse representation and the high resolution directory. Each sub-view image is captured with a different perspective and a different spectral range. The multi-perspective spectral image is obtain with one exposure.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of International PatentApplication No. PCT/CN2017/085148, filed on May 19, 2017 and entitled“METHOD AND SYSTEM FOR SNAPSHOT MULTI-SPECTRAL LIGHT FIELD IMAGING.” Theabove-referenced application is incorporated herein by reference in itsentirety.

TECHNICAL FIELD

The disclosure relates generally to a snapshot plenoptic imaging system,and more particularly, to a method and system for generating highresolution multi-spectral light fields.

BACKGROUND

A complete light field, which is also called plenoptic function, has sixdimensions. They are two spatial dimensions, two angular dimensions, onespectral dimension and one temporal dimension. A light field (LF)camera, also known as a plenoptic camera, captures light travelling inevery direction through a scene. That is, contrasted with a conventionalcamera, which records only light intensity, an LF camera captures boththe intensities and the directions of the light rays. This enablessophisticated data processing even after the image data is captured. Onecan for example virtually change focus or perspective, or estimate depthmaps from a single exposure.

However, the traditional LF cameras, for example, Lytro, Raytrix, etc.,using a micro-lens array, may only record four dimensions of the lightfield, i.e., two spatial and two angular dimensions. In this disclosure,we propose a new snapshot plenoptic imaging (SPI) system that is able tocapture 5D multi-spectral light fields (MLFs)—two spatial, two angularand one spectral dimensions, in a single exposure. This SPI systemutilizes spectral-coded catadioptric mirrors, and can generate 5D MLFswith a resolution of no more than 5 nm.

SUMMARY

One aspect of the present disclosure is directed to a method forgenerating high resolution multi-spectral light fields. The method mayinclude capturing a multi-perspective spectral image which includes aplurality of sub-view images; aligning and warping the sub-view imagesto obtain low resolution multi-spectral light fields; obtaining a highresolution dictionary and a low resolution dictionary; obtaining asparse representation based on the low resolution multi-spectral lightfields and the low resolution dictionary; and generating high resolutionmulti-spectral light fields with the sparse representation and the highresolution directory. Each sub-view image is captured with a differentperspective and a different spectral range. The multi-perspectivespectral image is obtained with one exposure.

Another aspect of the present disclosure is directed to a snapshotplenoptic imaging system for capturing images to generate highresolution multi-spectral light fields. The system may include aplurality of spectral-coded catadioptric mirrors, a digital camera andan image processing unit. Each catadioptric mirror is coated with adifferent spectral reflective coating. The digital camera is configuredto capture a multi-perspective spectral image. The multi-perspectivespectral image includes a plurality of sub-view images and each sub-viewimage is captured with a different perspective and a different spectralrange. The image processing unit is configured to align and warp thesub-view images to obtain low resolution multi-spectral light fields.Based on the low resolution multi-spectral light fields, high resolutionmulti-spectral light fields can be generated.

It is to be understood that the foregoing general description and thefollowing detailed description are exemplary and explanatory only, andare not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which constitute a part of this disclosure,illustrate several non-limiting embodiments and, together with thedescription, serve to explain the disclosed principles.

FIG. 1 is a schematic diagram for a snapshot plenoptic imaging system,consistent with exemplary embodiments of the present disclosure.

FIG. 2 (a) is graphical representation illustrating a data capturingunit of a snapshot plenoptic imaging system, FIG. 2 (b) is a photographof an exemplary prototype of the data capturing unit, consistent withexemplary embodiments of the present disclosure.

FIG. 3 is a diagram illustrating geometry of catadioptric mirrors,consistent with exemplary embodiments of the present disclosure.

FIG. 4 is a flow diagram illustrating a method of spectral imageregistration, consistent with exemplary embodiments of the presentdisclosure.

FIG. 5 is a flow diagram illustrating a method for spectral signalrecovery, consistent with exemplary embodiments of the presentdisclosure.

FIG. 6 is flow diagram illustrating a method for generating highresolution multi-spectral light fields by a snapshot plenoptic imagingsystem, consistent with exemplary embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to exemplary embodiments, examplesof which are illustrated in the accompanying drawings. The followingdescription refers to the accompanying drawings in which the samenumbers in different drawings represent the same or similar elementsunless otherwise represented. The implementations set forth in thefollowing description of exemplary embodiments consistent with thepresent invention do not represent all implementations consistent withthe invention. Instead, they are merely examples of systems and methodsconsistent with aspects related to the invention.

In accordance with embodiments of the present disclosure, an SPI systemincluding a data capturing unit and a data processing unit is provided.The data capturing unit captures a multi-perspective spectral image andtransmits it to the data processing unit. The data processing unitperforms ray-wise distortion correction, spectral image registration andspectral signal recovery. The ray-wise distortion correction correctsview distortions in the multi-perspective spectral image. The spectralimage registration is to align and warp sub-view images of themulti-perspective spectral image to generate low resolution MLFs. Thespectral signal recovery is to generate high resolution MLFs based onthe low resolution MLFs.

1. System Overview

FIG. 1 shows an exemplary SPI system 100 in accordance with anembodiment of the present disclosure. The system 100 may include anumber of components, some of which may be optional. In someembodiments, the system 100 may include many more components than thoseshown in FIG. 1. However, it is not necessary that all of thesecomponents be shown in order to disclose an illustrative embodiment.

As shown in FIG. 1, the system 100 may include a data capturing unit 200and a data processing unit 300. The data capturing unit 200 may includea digital camera 210 and a plurality of catadioptric mirrors 220. Thedata processing unit 300 may include a ray-wise distortion correctionunit 310, a spectral image registration unit 320 and a spectral signalrecovery unit 330.

2. Data Capturing Unit

FIG. 2 (a) is a graphical representation illustrating the data captureunit 200 in accordance with an embodiment of the present disclosure. Asshown in FIG. 2 (a), the data capturing unit 200 may include a camera210 and a plurality of catadioptric mirrors 220. In some embodiments,the camera 210 may be a commercially available digital single-lensreflex (DSLR) camera. The plurality of catadioptric mirrors 220 mayinclude 9 spectral-coded catadioptric mirrors. Each of the catadioptricmirrors may include a different spectral reflective coating, so thateach catadioptric mirror reflects light in a different spectral range.The catadioptric mirrors may have a spherical surface. The 9catadioptric mirrors may be mounted to acrylic lens holders in a 3×3array. The DSLR camera may be placed 2 meters away from the catadioptricmirrors and connected to a computer via a USB cable. FIG. 2 (b) shows aphotograph of an exemplary data capturing unit 200. Configurationparameters of an exemplary data capturing unit are listed in Table 1.

TABLE 1 Mirror Type Spherical Material BK7 Glass Radius 32.5 mm Baseline70.0 mm Coating Type Spectral reflective coating Spectral Range 400nm-700 nm Sensor Resolution 5760 × 3840 Field-of-View >100° SpectralResolution <10 nm MLF Resolution 1000 × 1000 × 3 × 3 × 30

2.1 Data Capturing Method

As shown in FIG. 2 (b), when capturing images, a scene of interest maybe placed between the DSLR camera and the catadioptric mirrors. In someembodiments, the scene is placed behind the camera. The DSLR cameracaptures one multi-perspective image at one exposure with the lightreflected by the catadioptric mirrors. In some embodiments, thecatadioptric mirrors are spectral-coded and placed in a 3×3 array. Eachmulti-perspective image includes 3×3 sub-view images. Receivingreflected lights from the spectral-coded catadioptric mirrors, each ofthe sub-view images is captured with a different perspective and adifferent spectral range. The captured multi-perspective image may betransmitted by a USB cable to the data processing unit 300.

3. Data Processing Unit

As shown in FIG. 1, the data processing unit 300 includes a ray-wisedistortion correction unit 310, a spectral image registration unit 320and a spectral signal recovery unit 330.

3.1 Ray-Wise Distortion Correction

The use of catadioptric mirrors provides a wide field of view (FOV), andalso causes strong view distortions in the captured images. Each of themulti-perspective image includes 9 distorted sub-view images fromdifferent perspectives. To correct the distortions in the sub-viewimages with different perspectives, a ray-wise distortion correctionmethod can be applied by analyzing reflectance geometry of thecatadioptric mirrors and re-projecting each pixel (ray) to a virtualfocal plane. The sub-view images present non-single view projections ofthe scene, since each of the sub-view image is captured at a differentperspective. Since baselines between any two adjacent sub-view imagesare the same, the distorted sub-view images may be re-projected by usinggeometric modeling as illustrated in FIG. 3. A baseline is the distancebetween adjacent sub-view images in pixel, usually measured in unit ofmm.

In FIG. 3, a spherical mirror of radius r is located at the origin O,and a camera aims at the sphere at a distance l. A ray of an incidentangle θ_(i) is reflected by the spherical surface with an angle ofreflectance θ_(r). In a 3D free space, all incident rays with the sameθ_(i) are re-projected to a bundle of rays with θ₁′ by a virtual cameraplaced at l′ on the axis. Due to the fact that the camera is very faraway from the mirrors: l>>r, the virtual camera position can beconsidered approximately as a constant within a desired FOV. Thus,tan θ_(i)·(l−r sin γ)=r·cos γwhere γ is the direction of surface normal. For any given θ_(i), γ canbe obtained as:

$\gamma = {\sin^{- 1}\left( \frac{{{l \cdot \tan^{2}}\theta_{i}} + \sqrt{{r^{2}\tan^{4}\theta_{i}} - {l^{2}\tan^{2}\theta_{i}} + r^{2}}}{r\left( {1 + {\tan^{2}\theta_{i}}} \right)} \right)}$Based on the law of reflection, the equivalent incident θ_(i)′ and l′can be obtained as follows:

θ_(i)^(′) = θ_(i) − 2 γ + π$l^{\prime} = {{r\;\sin\;\gamma} - \frac{r\;\cos\;\gamma}{\tan\;\theta_{i}^{\prime}}}$

A virtual focal plane can be defined by the position of the virtualcamera, i.e., l′. According to the above-discussed equations, setting anFOV as 100°, the equivalent incident θ_(i)′ and l′ can be obtained forall incident rays. By re-projecting each pixel (ray) to the virtualfocal plane, the sub-view image from one catadioptric mirror can becorrected from view distortions. The geometric modeling of a singlecatadioptric mirror can be simply extended to a mirror array due to themirror's rotational symmetry of its spherical surface. In someembodiments, the distance from the camera to a catadioptric mirror/maynot be a constant for all mirrors in a planar array configuration.

2.2 Spectral Image Registration

The multi-perspective image with the corrected sub-view images istransmitted to the spectral image registration 320 for spectral imageregistration. The objective of the spectral image registration is toalign and warp the sub-view images to any perspective so that a datacubeof a multi-spectral light field can be constructed on each perspective.In other words, for each particular sub-view image, align and warp theother sub-view images to the particular sub-view image. Then on eachperspective, a datacube, which includes a light field of a plurality ofdifferent spectral bands can be constructed. Here, a band refers to asection of light spectrum. In some embodiments, a multi-perspectiveimage may include 9 different perspectives and 9 different spectralbands. By performing the spectral image registration on themulti-perspective image, 9 multi-spectral datacubes, each of whichincludes 9 spectral bands, are constructed on each perspectiverespectively. These MLFs included in these datacubes are called lowresolution MLFs in this disclosure.

Traditional image registration methods, such as, patch-based stereo,optical flow, etc., fail to register spectral images, as the spectralimages share no intensity consistency. This disclosure presents aspectral image registration method to obtain disparity maps by utilizingboth a cross-spectrum feature descriptor and depth from defocustechnique. This method can align spectral images and generate highquality disparity maps, and warp the sub-view images to each other basedon the disparity maps. FIG. 4 shows a flow diagram illustrating themethod of spectral image registration in accordance with an exemplaryembodiment of the present disclosure.

2.2.1 Cross-Spectrum Feature Descriptor

A feature descriptor is a type of feature representation chosen to standfor a feature in image processing. In this disclosure, the robustselective normalized cross correlation (RSNCC), is adopted as across-spectrum feature descriptor. RSNCC is a robust matching metric forspectral images.

At step 401, by using RSNCC, pixel correspondence among the sub-viewimages can be established. The detailed computational process may beexplained in the following equations. Denoting a central view as I₁ andits neighboring view as I₂, a matching cost function E_(rsncc) can becomputed as:

E_(rsncc)(d, p) = f(1 − NCC(I₁, I₂, d, p)) + f(1 − NCC(M[I₁], M[I₂], d, p))  f(x) = −log (e^(−x) + e^(x − 2))$\mspace{20mu}{{{NCC}\left( {I_{1},I_{2},d,p} \right)} = \frac{\left( {{I_{1}(p)} - \overset{\_}{I_{1}(p)}} \right) \cdot \left( {{I_{2}\left( {p + d} \right)} - \overset{\_}{I_{2}\left( {p + d} \right)}} \right)}{{{{I_{1}(p)} - \overset{\_}{I_{1}(p)}}} \cdot {{{I_{2}\left( {p + d} \right)} - \overset{\_}{I_{2}\left( {p + d} \right)}}}}}$Here, d is a pixel disparity, p is a pixel position, M[I₁] and M[I₂]stand for gradient magnitudes of images I₁ and I₂. I₁(p) and I₂(p+d) arepixels' feature vectors in path p in I₁ and patch p+d in I₂respectively. I₁(p) is the average value of a patch of pixels centeredat p. NCC(I₁, I₂, d, p) is the normalized cross correlation between thepatch center at p in I₁ and patch p+d in I₂ in the intensity or colorspace. It represents structure similarity of the two patches underfeature I even if the two patches are transformed in color and geometrylocally. ƒ( ) is the robust selective function to help remove outliers.The outliers include structure divergence caused by shadow, highlight,etc. The robust selective function is effective in image matching andmakes RSNCC continuous and solvable by continuous optimization. Thecomputation results a matching function E_(rsncc) representing the pixelcorrespondence among the sub-view images.

2.2.2 Depth from Defocus

At step 402, differences in the depth of views (DOVs) of sub-view imagescan be estimated by a depth from defocus technique. The depth fromdefocus technique is a method that estimates a scene depth from a stackof refocused images with a shallow DOV. The depth from defocus methodcan be performed by fusing a multi-spectral focal stack. Themulti-spectral focal stack, including a plurality of views: I₁, I₂, . .. can be generated by refocusing the multi-spectral light field at aserial of depths. Then a patch-wise refocusing energy metric can becomputed as:

${E_{rf}\left( {d,p} \right)} = {- \frac{M\left\lbrack {I_{d}\left( {d,p} \right)} \right\rbrack}{\sum\limits_{i \in U}{M\left\lbrack {I_{d}\left( {d,i} \right)} \right\rbrack}}}$Here, M[ ] stands for a gradient magnitude of an image, U is a smallpatch centered at p.

2.2.3 Optimization

At step 403, a disparity map for each perspective can be obtained byoptimizing an energy function. The energy function utilizes both resultsfrom RSNCC and the depth from defocus, and can be constructed asfollowing:E(d,p)=α·E _(rsncc)(d,p)+β·E _(rf)(d,p)+P(d,p)Here, α and β are weight coefficients, and P(d,p) is a smooth penalty.The disparity map can be defined as:

${D_{map}(p)} = {\underset{d}{\arg\;\min}{\sum{E\left( {d,p} \right)}}}$This is an optimization problem, and can be solved by graph cutsalgorithm. “Graph cuts” is a type of algorithm used to solve a varietyof energy minimization problems which employ a max-flow/min-cutoptimization. At step 404, other sub-view are warped to the central viewusing the disparity map D_(map)(p), so that a multi-spectral datacube atthe central view can be obtained. At step 405, the same registration andwarping process are repeated for every perspective, and 9 multi-spectraldatacubes can be obtained. Accordingly, the low resolution MLFs can beobtained at step 405.

2.3 Spectral Signal Recovery

The low resolution MLFs have a low spectral resolution, including only 9spectral bands. Traditional SPI solutions are also limited to eitherspectral or spatial resolution. To overcome these limitations andincrease spectral resolution, the SPI system, in some embodiments, canadopt a sparse reconstruction algorithm to generate high resolutionmulti-spectral light fields. The sparse reconstruction algorithm is alearning-based reconstruction algorithm, which is used for acquiring,representing and compressing high-dimensional signals. Themulti-spectral signals can be sparsely represented under anover-complete dictionary, and the spectral sparsity of themulti-spectral signals allows compressed sensing of the signals.Compressed sensing (also known as compressive sensing, compressivesampling, or sparse sampling) is a signal processing technique forefficiently acquiring and reconstructing a signal, by finding solutionsto underdetermined linear systems. This is based on the principle that,through optimization, the sparsity of a signal can be exploited torecover it from far fewer samples than required by the Shannon-Nyquistsampling theorem. Representing a signal involves the choice of theso-called “dictionary”, which is a set of elementary signals used todecompose the signal. When the dictionary forms a basis, the signal canbe uniquely represented as a linear combination of the elementarysignals. “Over-complete” indicates that the number of basis is more thanthe dimensionality of the signals. Finding a sparse representation of asignal in a given over-complete dictionary is to find the correspondingsparse coefficients in the linear combination of the elementary signals.

The spectral-coded catadioptric mirrors are able to sense the 5Dplenoptic function compressively. Therefore, the high resolution MLFscan be generated computationally by using a spectral dictionary. Aspectral dictionary can be learned from a publicly availablemulti-spectral image database. In some embodiments, the database can beretrieved from http://www.cs.columbia.edu/CAVE/databases/multispectral/.The database consists of 32 scenes, and each scene includes fullspectral resolution reflectance data from 400 nm to 700 nm at 10 nmsteps (31 bands total). With the obtained dictionary, the sparsereconstruction algorithm is able to improve the low resolution MLFs to ahigh resolution MLFs from 30 nm to no more than 10 nm.

The obtained low resolution MLFs are input to the spectral signalrecovery unit 330. FIG. 5 shows a flow diagram illustrating the methodfor generating high resolution MLFs in accordance with the exemplaryembodiment of the present disclosure.

At step 501, two dictionaries can be obtained from the publiclyavailable multi-spectral database: a high resolution dictionary D_(h)and a corresponding low dictionary D_(l). Both are used to generate thehigh resolution multi-spectral high fields. The detailed method can beexplained as following.

Given a training set of high resolution spectral signals: h₁, h₂, . . ., which is obtained from the publicly available database, twodictionaries can be obtained by solving a l₁ norm optimization problem.The l₁ norm optimization problem is a technique commonly used incompressed sensing. In one exemplary embodiment, the sparsereconstruction algorithm can be represented as following:

$D^{*} = {{\underset{D^{\prime}}{\arg\;\min}\frac{1}{n}{\sum\limits_{i = 1}^{n}{\frac{1}{2}{{h_{i}^{\prime} - {D^{\prime} \cdot \beta_{i}}}}_{2}^{2}}}} + {\lambda{\beta_{i}}_{1}}}$Here β_(i) is a sparse coefficient, and l|β_(i)∥₁ is a term used topenalize β_(i) for not being sparse. The first term is a reconstructionterm which tries to force the algorithm to provide a good representationof h₁′ and the second term is a sparsity penalty which forces therepresentation of h_(i)′ to be sparse. λ is a scaling constant todetermine the relative importance of these two contributions.

$D^{\prime} = {{\begin{pmatrix}D_{h} \\D_{l}\end{pmatrix}\mspace{31mu} h_{i}^{\prime}} = \begin{pmatrix}h_{i} \\{S \cdot h_{i}}\end{pmatrix}}$where S is a known sampling matrix. Denoting spectral reflectancefunction of the 9 mirrors as R₁, R₂, . . . R₉ and Bayer filter's quantumefficiency as B_(R), B_(G), B_(B), the sampling matrix S is composed as:

$S = \begin{bmatrix}{{B_{R}\left( \lambda_{\min} \right)}*{R_{1}\left( \lambda_{\min} \right)}} & \ldots & {{B_{R}\left( \lambda_{\max} \right)}*{R_{1}\left( \lambda_{\max} \right)}} \\\vdots & \ddots & \vdots \\{{B_{R}\left( \lambda_{\min} \right)}*{R_{9}\left( \lambda_{\min} \right)}} & \ldots & {{B_{R}\left( \lambda_{\max} \right)}*{R_{9}\left( \lambda_{\max} \right)}} \\{{B_{G}\left( \lambda_{\min} \right)}*{R_{1}\left( \lambda_{\min} \right)}} & \ldots & {{B_{G}\left( \lambda_{\max} \right)}*{R_{1}\left( \lambda_{\max} \right)}} \\\vdots & \ddots & \vdots \\{{B_{G}\left( \lambda_{\min} \right)}*{R_{9}\left( \lambda_{\min} \right)}} & \ldots & {{B_{G}\left( \lambda_{\max} \right)}*{R_{9}\left( \lambda_{\max} \right)}} \\{{B_{B}\left( \lambda_{\min} \right)}*{R_{1}\left( \lambda_{\min} \right)}} & \ldots & {{B_{B}\left( \lambda_{\max} \right)}*{R_{1}\left( \lambda_{\max} \right)}} \\\vdots & \ddots & \vdots \\{{B_{B}\left( \lambda_{\min} \right)}*{R_{9}\left( \lambda_{\min} \right)}} & \ldots & {{B_{B}\left( \lambda_{\max} \right)}*{R_{9}\left( \lambda_{\max} \right)}}\end{bmatrix}$Here, λ_(min) and λ_(max) stand for a minimum wavelength and a maximumwavelength of an interested spectrum. Two dictionaries D_(h) and D_(l)are co-trained so that the high and low resolution multi-spectralsignals (h_(i) and S·h_(i)) have the same sparse representation,{circumflex over (β)}, a set of β_(i), with respect to the dictionaries.

At step 502, a sparse representation, {circumflex over (β)}, can beobtained based on the two dictionaries. As discussed previously, the SPIsystem senses the compressed MLFs. Low resolution (9 bands) MLFs can beobtained by performing the distortion correction and image registration.For each of the low resolution spectral signal l in the light fields,its corresponding sparse representation under the low resolutiondictionary D_(l) can be solved using LASSO method. LASSO stands forabsolute shrinkage and selection operator, and is a regression analysisthat performs both variable selection and regulation in order to enhancethe prediction accuracy and interpretability of the statistical model itproduces. Accordingly, the sparse representation {circumflex over (β)}can be solved by the following function:

$\hat{\beta} = {{\arg\;{\min\limits_{\beta}{\frac{1}{2}{{l - {D_{l} \cdot \beta}}}_{2}^{2}\mspace{31mu}{s.t.\mspace{14mu}{\beta }_{1}}}}} < k}$Here, k is a sparse parameter.

Since the high and low resolution spectral signals have the same sparserepresentation {circumflex over (β)}, with respect to the dictionaries,the high resolution spectral signal h can be generated using the sparserepresentation and the high resolution dictionary as following:h=D _(h)·{circumflex over (β)}Accordingly, at step 503, the high resolution MLFs can be generated byusing the sparse representation and the high resolution dictionary.

The SPI system has the following advantages: 1. It is the first singlesensor solution to capture a 5D plenoptic function. Previous solutionsemploy either a hybrid sensor system or a sensor array to acquire highdimensional light fields. These architectures are bulky in size andexpensive. The SPI system in this disclosure uses a single commercialDSLR camera and an optical unit (mirror array) which is portable andless expensive. The single sensor property requires no synchronization,while the synchronization can be very challenging for multiple sensorarrays.

2. The snapshot property of the SPI system provides a potential foracquiring temporal dimensions. Unlike scanning-based spectral imagingtechniques, the SPI records the compressed MLFs in a single exposure,and is applicable for capturing dynamic scene or even multi-spectrallight field videos.

3. Due to the rotational symmetry of spherical mirrors, there is no needto calibrate the orientation of any individual mirror, which isconvenient for users to increase the number of mirrors. Also, thespectral range of the SPI system can be extended by replacing some ofthe mirrors. In some embodiments, the SPI system measures only visiblespectral; and in some other embodiments, the system can acquire nearinfra-red and ultra violet information by using additionalspectral-coded mirrors or auxiliary flash devices.

FIG. 6 illustrates a method for generating high resolution MLFs by theSPI, in accordance with an embodiment of the present disclosure. At step601, a multi-perspective spectral image is captured; at step 602, thecaptured multi-perspective spectral image is corrected from a viewdistortion by the ray-wise distortion correction method; at step 603,the sub-view images of the multi-perspective spectral image are alignedand warped to generate low resolution MLFs; and at step 604, the highresolution MLFs are obtained based on the low resolution MLFs and thehigh and low resolution dictionaries.

The various modules, units, and components described above can beimplemented as an Application Specific Integrated Circuit (ASIC); anelectronic circuit; a combinational logic circuit; a field programmablegate array (FPGA); a processor (shared, dedicated, or group) thatexecutes code; or other suitable hardware components that provide thedescribed functionality. The processor can be a microprocessor providedby from Intel, or a mainframe computer provided by IBM.

Note that one or more of the functions described above can be performedby software or firmware stored in memory and executed by a processor, orstored in program storage and executed by a processor. The software orfirmware can also be stored and/or transported within anycomputer-readable medium for use by or in connection with an instructionexecution system, apparatus, or device, such as a computer-based system,processor-containing system, or other system that can fetch theinstructions from the instruction execution system, apparatus, or deviceand execute the instructions. In the context of this document, a“computer-readable medium” can be any medium that can contain or storethe program for use by or in connection with the instruction executionsystem, apparatus, or device. The computer readable medium can include,but is not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus or device,a portable computer diskette (magnetic), a random access memory (RAM)(magnetic), a read-only memory (ROM) (magnetic), an erasableprogrammable read-only memory (EPROM) (magnetic), a portable opticaldisc such a CD, CD-R, CD-RW, DVD, DVD-R, or DVD-RW, or flash memory suchas compact flash cards, secured digital cards, USB memory devices,memory sticks, and the like.

The invention described and claimed herein is not to be limited in scopeby the specific preferred embodiments disclosed herein, as theseembodiments are intended as illustrations of several aspects of theinvention. Indeed, various modifications of the invention in addition tothose shown and described herein will become apparent to those skilledin the art from the foregoing description. Such modifications are alsointended to fall within the scope of the appended claims.

What is claimed is:
 1. A method for generating high resolutionmulti-spectral light fields, comprising: capturing a multi-perspectivespectral image with one exposure, wherein the multi-perspective spectralimage includes a plurality of sub-view images and each sub-view image iscaptured with a different perspective and a different spectral range;aligning and warping the sub-view images to obtain low resolutionmulti-spectral light fields; obtaining a high resolution dictionary anda low resolution dictionary; obtaining a sparse representation based onthe low resolution multi-spectral light fields and the low resolutiondictionary; and generating high resolution multi-spectral light fieldswith the sparse representation and the high resolution directory.
 2. Themethod of claim 1, wherein the high resolution directory and the lowresolution directory are co-trained.
 3. The method of claim 1, furthercomprising: using a robust selective normalized cross correlation toobtain a pixel correspondence among the sub-view images.
 4. The methodof claim 1, further comprising: estimating differences in depth of viewsof the sub-view images by a depth from defocus technique.
 5. The methodof claim 1, wherein the sub-view images are warped to each other byusing a disparity map.
 6. The method of claim 1, further comprising:correcting view distortions in the multi-perspective spectral image byanalyzing reflectance geometry and re-projecting each pixel to a virtualfocal plane.
 7. The method of claim 1, wherein the low resolutionmulti-spectral light fields include nine spectral bands.
 8. The methodof claim 1, wherein the high resolution multi-spectral light fields havea resolution of no more than 5 nm.
 9. The method of claim 1, furthercomprising: capturing the multi-perspective spectral image using adigital camera and a plurality of spectral-coded catadioptric mirrors.10. The method of claim 9, wherein each catadioptric mirror is coatedwith a different spectral reflective coating.
 11. The method of claim 9,wherein the catadioptric mirrors form a 3×3 array.
 12. The method ofclaim 9, wherein each catadioptric mirror has a spherical surface.
 13. Asnapshot plenoptic imaging system for capturing images to generate highresolution multi-spectral light fields, the system comprising: aplurality of spectral-coded catadioptric mirrors, wherein eachcatadioptric mirror is coated with a different spectral reflectivecoating; a digital camera configured to capture a multi-perspectivespectral image, wherein the multi-perspective spectral image includes aplurality of sub-view images and each sub-view image is captured with adifferent perspective and a different spectral range; and an imageprocessing unit configured to align and warp the sub-view images toobtain low resolution multi-spectral light fields and generate highresolution multi-spectral light fields based on the low resolutionmulti-spectral light fields.
 14. The system of claim 13, wherein theimage processing unit is configured to obtain a high resolutiondictionary and a low resolution dictionary; obtain a sparserepresentation based on the low resolution multi-spectral light fieldsand the low resolution dictionary; and generate the high resolutionmulti-spectral light fields with the sparse representation and the highresolution directory.
 15. The system of claim 13, wherein the highresolution directory and the low resolution directory are co-trained.16. The system of claim 13, wherein the image processing unit isconfigured to use a robust selective normalized cross correlation toobtain a pixel correspondence among the sub-view images.
 17. The systemof claim 13, wherein the image processing unit is configured to estimatedifferences in depth of views of the sub-view images by a depth fromdefocus technique.
 18. The system of claim 13, wherein the sub-viewimages are warped to each other by using a disparity map.
 19. The systemof claim 13, wherein the image processing unit is configured to correctview distortions in the multi-perspective spectral image by analyzingreflectance geometry and re-projecting each pixel to a virtual focalplane.
 20. The system of claim 13, wherein the low resolutionmulti-spectral light fields include nine spectral bands.
 21. The systemof claim 13, wherein the high resolution multi-spectral light fieldshave a resolution of no more than 5 nm.
 22. The system of claim 13,wherein the catadioptric mirrors form a 3×3 array on a plane.
 23. Thesystem of claim 13, wherein each catadioptric mirror has a sphericalsurface.